Papers with knowledge distillation methods
Meta-Learning Adaptive Knowledge Distillation for Efficient Biomedical Natural Language Processing (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods have been proposed to reduce the size of large models for biomedical natural language processing tasks. |
| Approach: | They propose a meta-learning approach which adaptively learns parameters that enable optimal rate of knowledge exchange between teacher and student models from the distillation data during knowledge distillation. |
| Outcome: | The proposed method improves the performance of knowledge distillation methods on two biomedical natural language processing tasks. |
Distilling Text Style Transfer With Self-Explanation From LLMs (2024.naacl-srw)
Copied to clipboard
| Challenge: | Text Style Transfer (TST) aims to alter the style of text while preserving its core content. |
| Approach: | They propose a framework that leverages large language models alongside chain-of-thought prompting to facilitate TST. |
| Outcome: | The proposed framework surpasses supervised fine-tuning and knowledge distillation methods in low-resource settings. |
Towards Developing a Multilingual and Code-Mixed Visual Question Answering System by Knowledge Distillation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for training language-vision models only consider monolingual learning, especially English. |
| Approach: | They propose to extend an English language-vision model into a multilingual and code-mixed model by using knowledge distillation techniques. |
| Outcome: | The proposed model outperforms existing models on multilingual and code-mixed VQA datasets on eleven languages. |
DialoKG: Knowledge-Structure Aware Task-Oriented Dialogue Generation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent research focused on knowledge distillation methods where the underlying relationship between the facts in a knowledge base is not effectively captured. |
| Approach: | They propose a novel task-oriented dialogue system that effectively incorporates knowledge into a language model by using structural information of a knowledge graph. |
| Outcome: | The proposed system views relational knowledge as a knowledge graph and introduces (1) a structure-aware knowledge embedding technique, and (2) a Knowledge graph-weighted attention masking strategy to facilitate the system selecting relevant information during the dialogue generation. |
Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) face high computational demands at inference time due to high computational costs. |
| Approach: | They propose a cost-effective and high-throughput solution for large language models . PGKD distills the knowledge of LLMs into smaller, task-specific models based on teacher-student knowledge distillation . |
| Outcome: | PGKD outperforms BERT-based models and other knowledge distillation methods on multi-class classification datasets. |
Okay, Let’s Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent work shows that generative large language models (LLMs) can be used to solve cross-document coreference problems. |
| Approach: | They propose rationale-oriented event clustering and knowledge distillation methods for event coreference scoring that leverage enriched information from the FTRs for improved CDCR. |
| Outcome: | The proposed model achieves SOTA B3 F1 on the ECB+ and GVC corpora without additional annotation or expensive document clustering. |
Extremely Small BERT Models from Mixed-Vocabulary Training (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods cannot be directly applied to train student models with reduced vocabulary and embedding dimensions. |
| Approach: | They propose a method to align teacher and student embeddings via mixed-vocabulary training. |
| Outcome: | The proposed method compresses BERT-LARGE to a task-agnostic model with smaller vocabulary and hidden dimensions, which is an order of magnitude smaller than other distilled models. |
Knowledge Distillation for Multilingual Unsupervised Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Unsupervised neural machine translation (UNMT) can only translate between a single language pair and cannot produce translation results for multiple language pairs at the same time. |
| Approach: | They propose a method to translate between 13 languages using a single encoder and a decoder . they propose two knowledge distillation methods to further enhance multilingual UNMT performance . |
| Outcome: | The proposed method improves translation performance for all languages using multilingual data. |
SimulSpeech: End-to-End Simultaneous Speech to Text Translation (2020.acl-main)
Copied to clipboard
| Challenge: | SimulSpeech is an end-to-end simultaneous speech to text translation system . conventional approaches to simultaneous speech translation divide the translation process into two stages . |
| Approach: | They develop an end-to-end simultaneous speech to text translation system which translates speech in source language to text in target language concurrently. |
| Outcome: | The proposed system achieves reasonable BLEU scores and lower delay compared to full-sentence translation model. |
MT-PATCHER: Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown their strong ability in the field of machine translation, yet they suffer from high computational cost and latency. |
| Approach: | They propose a framework which transfers knowledge from LLMs to existing MT models in a selective, comprehensive and proactive manner. |
| Outcome: | The proposed framework transfers knowledge from LLMs to existing MT models in a selective, comprehensive and proactive manner. |
Patient Knowledge Distillation for BERT Model Compression (D19-1)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT have proven to be highly effective for natural language processing tasks, but the high demand for computing resources hinders their application in practice. |
| Approach: | They propose to compress an original large model (teacher) into an equally-effective lightweight shallow network (student) Empirically, this translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy. |
| Outcome: | The proposed model reduces the computational cost of training models using the teacher model into a lightweight shallow network. |
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information. |
| Approach: | They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods. |
Adaptive Contrastive Knowledge Distillation for BERT Compression (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods for BERT implicitly learn discriminative student features by mimicking the teacher features. |
| Approach: | They propose a new knowledge distillation approach called adaptive contrastive knowledge distilling for BERT compression using hidden state features in BERT as explicit supervision to learn discriminative student features. |
| Outcome: | The proposed approach improves on multiple natural language processing tasks. |
ConNER: Consistency Training for Cross-lingual Named Entity Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing consistency training methods for named entity recognition (NER) are likely to violate the consistency hypothesis or focus on coarse-grain consistency. |
| Approach: | They propose a consistency training framework for cross-lingual named entity recognition that leverages unlabeled target-language data and dropout-based consistency training on labeled source-language datasets. |
| Outcome: | The proposed framework improves on translation-based consistency training on unlabeled target-language data and dropout-based consistent training on labeled source-language datasets. |
Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods rely on intermediate layer features and golden labels, which require aligned model architecture and labeled data respectively. |
| Approach: | They propose a general language model distillation method that performs two-stage word prediction distillation and vocabulary compression, which is simple and shows extremely strong performance. |
| Outcome: | The proposed method outperforms 25 state-of-the-art methods on the SuperGLUE benchmark, achieving an average score that surpasses the best method by 3%. |
StepER: Step-wise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods overlook the need for different reasoning abilities at different steps, hindering transfer in multi-step retrieval-augmented frameworks. |
| Approach: | They propose a method that uses step-wise supervision to align with evolving information and reasoning demands across stages. |
| Outcome: | The proposed method outperforms previous methods on multi-hop QA benchmarks with an 8B model achieving performance comparable to a 70B teacher model. |
When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large vision and language models have demonstrated remarkable performance in visual question answering tasks. |
| Approach: | They introduce a framework to optimize L-VLMs by leveraging unlabeled images . they conduct extensive experiments on four diverse VQA benchmarks . |
| Outcome: | The proposed framework improves L-VLMs on four visual question answering benchmarks. |